<html>
<head><meta charset="utf-8"><title>locally tracking `rustc` compile-times per-commit · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html">locally tracking `rustc` compile-times per-commit</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="179423662"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423662" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423662">(Oct 30 2019 at 11:54)</a>:</h4>
<p>reading over <a href="https://github.com/rust-lang/rust/issues/65927" target="_blank" title="https://github.com/rust-lang/rust/issues/65927">#65927</a>, I was idly wondering whether we could make it easier for developers to catch these regressions <em>locally</em>. Namely, <strong>without</strong> using <code>perf</code> or any other infrastructure.</p>



<a name="179423726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423726">(Oct 30 2019 at 11:55)</a>:</h4>
<p><span class="user-mention" data-user-id="119009">@eddyb</span> and I talked a bit about that -- in theory, this should've been caught by incremental tests, and I believe <span class="user-mention" data-user-id="124287">@mw</span> was surprised it wasn't</p>



<a name="179423753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423753">(Oct 30 2019 at 11:55)</a>:</h4>
<p>and that led me to thus potentially bonkers idea: Would it make any sense to track the historical completed build times of stage2 <code>rustc</code> for each commit, and then after each build that is a child of the <code>master</code> commit, compare the median values in the history, and print out the time delta...?</p>



<a name="179423754"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423754" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423754">(Oct 30 2019 at 11:55)</a>:</h4>
<p>and it's definitely possible to run perf locally, at least on Linux-based systems -- but that's sort of a non-answer</p>



<a name="179423803"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423803" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423803">(Oct 30 2019 at 11:56)</a>:</h4>
<p>(by <code>perf</code> I meant <code>perf.rlo</code>)</p>



<a name="179423831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423831">(Oct 30 2019 at 11:56)</a>:</h4>
<blockquote>
<p><span class="user-mention silent" data-user-id="119009">eddyb</span> and I talked a bit about that -- in theory, this should've been caught by incremental tests, and I believe <span class="user-mention silent" data-user-id="124287">mw</span> was surprised it wasn't</p>
</blockquote>
<p>the incremental tests check compile times?</p>



<a name="179423844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423844">(Oct 30 2019 at 11:57)</a>:</h4>
<p>No, but in this case the regression was "we stopped skipping codegen"</p>



<a name="179423848"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423848" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423848">(Oct 30 2019 at 11:57)</a>:</h4>
<p>heh</p>



<a name="179423852"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423852" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423852">(Oct 30 2019 at 11:57)</a>:</h4>
<p>which should've been caught</p>



<a name="179423856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423856">(Oct 30 2019 at 11:57)</a>:</h4>
<p>i see</p>



<a name="179423861"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423861" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423861">(Oct 30 2019 at 11:57)</a>:</h4>
<p>(but we apparently didn't have a test)</p>



<a name="179423873"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423873" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423873">(Oct 30 2019 at 11:57)</a>:</h4>
<p>still, the more general idea may have some value</p>



<a name="179423879"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423879" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423879">(Oct 30 2019 at 11:57)</a>:</h4>
<p>if we can implement it semi-robustly...</p>



<a name="179423943"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423943" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423943">(Oct 30 2019 at 11:58)</a>:</h4>
<p>I think part of the problem is that we have really high variance on CI, and locally there's maybe less variance but there's nothing to compare against reliably</p>



<a name="179423946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423946">(Oct 30 2019 at 11:58)</a>:</h4>
<p>(and in any case, it wouldn't help with this AFAIK)</p>



<a name="179423961"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423961" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423961">(Oct 30 2019 at 11:58)</a>:</h4>
<p>the thing to compare it against is the stored history , I'd say</p>



<a name="179423967"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423967" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423967">(Oct 30 2019 at 11:58)</a>:</h4>
<p>since you're almost never recompiling the compiler in a "perfect" incremental scenario</p>



<a name="179423990"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179423990" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179423990">(Oct 30 2019 at 11:58)</a>:</h4>
<p>Oh I suppose you're right: incremental compiles mean that the times are useless</p>



<a name="179424007"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424007" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424007">(Oct 30 2019 at 11:59)</a>:</h4>
<p>okay so comparing the stage2 <code>rustc</code> times is no good</p>



<a name="179424067"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424067" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424067">(Oct 30 2019 at 11:59)</a>:</h4>
<p>an alternative might be to make a compile-time benchmark suite part of <code>x.py test</code>, and have that store local history in the build directory</p>



<a name="179424163"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424163" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424163">(Oct 30 2019 at 12:00)</a>:</h4>
<p>where the benchmark suite would presumably be some subset of perf.rlo</p>



<a name="179424177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424177">(Oct 30 2019 at 12:00)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> are you expecting devs to use stage2?</p>



<a name="179424178"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424178" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424178">(Oct 30 2019 at 12:00)</a>:</h4>
<p>sure, we can do something like that</p>



<a name="179424184"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424184" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424184">(Oct 30 2019 at 12:00)</a>:</h4>
<p>this is not quite as nice as tracking the build time for <code>rustc</code>, since that's a build artifact that people are always going to need ...</p>



<a name="179424196"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424196" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424196">(Oct 30 2019 at 12:01)</a>:</h4>
<p><span class="user-mention" data-user-id="126931">@centril</span> this was one of the flaws in my plan</p>



<a name="179424210"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424210" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424210">(Oct 30 2019 at 12:01)</a>:</h4>
<p><span class="user-mention" data-user-id="126931">@centril</span> but a test benchmark suite would not need to rely on a stage2 rustc</p>



<a name="179424220"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424220" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424220">(Oct 30 2019 at 12:01)</a>:</h4>
<p>Yes, I think you'll have pitchforks at the gates if you make everyone use stage2 ;)</p>



<a name="179424230"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424230" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424230">(Oct 30 2019 at 12:01)</a>:</h4>
<p>I think it could work in theory, though we've not had super reliable measurements historically for e.g. wall time</p>



<a name="179424277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424277">(Oct 30 2019 at 12:02)</a>:</h4>
<p>and instruction counts I'm not sure how to record on windows/macOS</p>



<a name="179424279"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424279" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424279">(Oct 30 2019 at 12:02)</a>:</h4>
<p>maybe we wouldn't report anything at all unless the wall-clock time was truly absurd</p>



<a name="179424286"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424286" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424286">(Oct 30 2019 at 12:02)</a>:</h4>
<p>i.e. a 10x regression or something</p>



<a name="179424297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424297">(Oct 30 2019 at 12:02)</a>:</h4>
<p>we could, sure -- I personally think it would not be helpful, i.e., in practice doesn't catch enough</p>



<a name="179424324"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424324" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424324">(Oct 30 2019 at 12:03)</a>:</h4>
<p>Personally I'm happy that there's a central perf.rl.o cause I'm one of those egghead type-theory types that knows little about perf and wouldn't be able to handle local perf testing in a reliable way</p>



<a name="179424400"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424400" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424400">(Oct 30 2019 at 12:04)</a>:</h4>
<p>basically a 10x regression would time out CI reliably</p>



<a name="179424420"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424420" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424420">(Oct 30 2019 at 12:04)</a>:</h4>
<blockquote>
<p>we could, sure -- I personally think it would not be helpful, i.e., in practice doesn't catch enough</p>
</blockquote>
<p>you mean because a 10x regression, while filtering noise, is too high a threshold in general? E.g. for <a href="https://github.com/rust-lang/rust/issues/65927" target="_blank" title="https://github.com/rust-lang/rust/issues/65927">#65927</a>, only half of the benchmarks regressed at all, and of those, the median regression was like 3x or so?</p>



<a name="179424457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424457">(Oct 30 2019 at 12:05)</a>:</h4>
<p>maybe part of the problem here is that I'm coupling two distinct things: Gathering the data itself, and reporting it by default during build, and/or gating on it</p>



<a name="179424484"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424484" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424484">(Oct 30 2019 at 12:05)</a>:</h4>
<p>i.e. it might be interesting to have the information captured locally and some easy way to query it</p>



<a name="179424714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424714">(Oct 30 2019 at 12:09)</a>:</h4>
<p>we can definitely start capturing the information (e.g., run cargo with -Ztiming and save that off) -- I suspect most people wouldn't really find this helpful, but we can do it</p>



<a name="179424946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424946">(Oct 30 2019 at 12:12)</a>:</h4>
<p>I think the trickiest thing, in terms of my ideal setup, would be making sure the data-sets are properly separated. In particular, its not enough to treat the commit-id as a unique key to identify the data-set; I think you'd probably want commit-id + <code>git diff</code> output, or something.</p>



<a name="179424973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179424973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179424973">(Oct 30 2019 at 12:12)</a>:</h4>
<p>that, or only do the tracking if the <code>git diff</code> is empty</p>



<a name="179425003"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179425003" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179425003">(Oct 30 2019 at 12:13)</a>:</h4>
<p>(which is probably the best way to go. At least for me, the point where I commit is usually the point where I'm saying "okay I finally got this into a plausible and buildable state.)</p>



<a name="179425435"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179425435" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179425435">(Oct 30 2019 at 12:18)</a>:</h4>
<p>yes, I think that'd be fine, some similar heuristic (maybe enough, then, to just save by commit id and a bit of "clean git diff")</p>



<a name="179425459"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179425459" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179425459">(Oct 30 2019 at 12:19)</a>:</h4>
<p>which would overwrite by commit id so if you did end up building in clean mode then you'd be fine</p>



<a name="179425608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179425608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179425608">(Oct 30 2019 at 12:21)</a>:</h4>
<p>the more I reflect on this, I think the reason I want it is that usually by the time I realize I want to see the data here, I've already done several builds, and its expensive to redo them just to get timing history. Maybe I <em>do</em> want to measure the stage1 build times after all, too... hmm.</p>



<a name="179425672"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179425672" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179425672">(Oct 30 2019 at 12:22)</a>:</h4>
<p>well, it sounds like something I can play with locally at first</p>



<a name="179425808"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179425808" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179425808">(Oct 30 2019 at 12:24)</a>:</h4>
<p>I think tracking build times is something we should be doing more (in general, not just in rustc) so we can have hopefully happier users -- right now compile times are mostly a gut feeling</p>



<a name="179425816"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179425816" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179425816">(Oct 30 2019 at 12:24)</a>:</h4>
<p>but _how_ to do that is another story :)</p>



<a name="179426233"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179426233" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179426233">(Oct 30 2019 at 12:30)</a>:</h4>
<blockquote>
<p><span class="user-mention silent" data-user-id="119009">eddyb</span> and I talked a bit about that -- in theory, this should've been caught by incremental tests, and I believe <span class="user-mention silent" data-user-id="124287">mw</span> was surprised it wasn't</p>
</blockquote>
<p>let's put it this way: it wouldn't be too hard write a test that makes sure that clean rebuilds completely skip codegen (for the given test). I would have thought that there already is a test like that but apparently there isn't. Meanwhile I remembered that the <a href="https://github.com/nikomatsakis/cargo-incremental/" target="_blank" title="https://github.com/nikomatsakis/cargo-incremental/">cargo-incremental</a> tool and the rust-icci infrastructure based on it were doing such tests (extensively so). But rust-icci was decommissioned because it didn't seem worth the maintenance effort. That's still true because perf.rlo also let us spot the regression just as quickly.</p>



<a name="179427278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179427278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179427278">(Oct 30 2019 at 12:46)</a>:</h4>
<p>part of what I want is CI in the loop so we catch at least <em>some</em> things before ever landing a PR</p>



<a name="179427365"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179427365" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179427365">(Oct 30 2019 at 12:47)</a>:</h4>
<p>one thing I was suggesting to <span class="user-mention" data-user-id="116122">@simulacrum</span> was that <em>if there is a cheap way to do it</em>, we could run some of the more stress-test-y benchmarks that only take a few seconds (like <code>inflate</code>... wait how is that that fast? is that old <code>inflate</code> or the new one? I remember it taking minutes o_O)</p>



<a name="179427437"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179427437" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179427437">(Oct 30 2019 at 12:48)</a>:</h4>
<p>so basically spend no more than 5 minutes per PR CI (<em>not</em> <code>@bors</code>) checking a few benchmarks and reporting back above a certain threshold, to suggest a try + perf run</p>



<a name="179427468"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179427468" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179427468">(Oct 30 2019 at 12:49)</a>:</h4>
<p><a href="/user_uploads/4715/VZsDL4HAuBnDKty6A_O4EQlc/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> we've made it faster over time</p>
<div class="message_inline_image"><a href="/user_uploads/4715/VZsDL4HAuBnDKty6A_O4EQlc/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/VZsDL4HAuBnDKty6A_O4EQlc/pasted_image.png"></a></div>



<a name="179427577"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179427577" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179427577">(Oct 30 2019 at 12:50)</a>:</h4>
<p>wow</p>



<a name="179427608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179427608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179427608">(Oct 30 2019 at 12:51)</a>:</h4>
<p>wait why is min/max confusing when in relative terms? I guess it's the %Δ between the absolute min/max</p>



<a name="179428527"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179428527" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179428527">(Oct 30 2019 at 13:02)</a>:</h4>
<p>hm, no, that's the min/max percent</p>



<a name="179428537"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179428537" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179428537">(Oct 30 2019 at 13:03)</a>:</h4>
<p>it's not anything fancy</p>



<a name="179428555"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179428555" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179428555">(Oct 30 2019 at 13:03)</a>:</h4>
<p>it's not "magnitude" min/max though</p>



<a name="179428686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179428686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179428686">(Oct 30 2019 at 13:04)</a>:</h4>
<p>that's what confused me, sorry, I expected to find "smallest delta" and "largest delta" not literally min/max of the signed percentage</p>



<a name="179428841"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179428841" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179428841">(Oct 30 2019 at 13:06)</a>:</h4>
<p>oh</p>



<a name="179428876"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/locally%20tracking%20%60rustc%60%20compile-times%20per-commit/near/179428876" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/locally.20tracking.20.60rustc.60.20compile-times.20per-commit.html#179428876">(Oct 30 2019 at 13:06)</a>:</h4>
<p>yeah</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>